✨ feat: add pod status checking to distinguish pending vs running jobs#51
Open
✨ feat: add pod status checking to distinguish pending vs running jobs#51
Conversation
…n_progress misclassification
…er and status of job is returned as unknown
- Extracted Celery and direct submission logic into helper functions. - Fixed a blocking bug by offloading Celery's to a thread pool executor. - Normalized generation to ensure consistent returns on failure. - Improved readability and maintainability by simplifying the status mapping logic.
Avoid blocking the event loop while waiting for Celery response timeout.
fMurugi
requested review from
WanjiruCate,
bedwards-ibm,
bglar,
cwachira and
fredotieno
as code owners
bglar
changed the title
Contributor
|
Deploying this to an openshift cluster to test... |
fredotieno
reviewed
May 5, 2026
fredotieno
reviewed
May 5, 2026
bglar
requested changes
May 5, 2026
Contributor
There was a problem hiding this comment.
@fMurugi use black and ruff to format changes introduced
Example for the gfmstudio/fine_tuning/core/kubernetes.py file:
black gfmstudio/fine_tuning/core/kubernetes.py && ruff check --select I --fix gfmstudio/fine_tuning/core/kubernetes.py
…tial-studio-core into feat/check-pod-phase
bglar
requested changes
May 5, 2026
WanjiruCate
reviewed
May 6, 2026
| Returns | ||
| ------- | ||
| str | ||
| The pod phase: 'Running', 'Pending', 'Succeeded', 'Failed', 'Unknown', or None if no pod found |
Contributor
There was a problem hiding this comment.
This is a great introduction. Thinking of easily maintaining this in the future, creating a reference schema class for these statuses would be ideal ensuring that you would only have one place to change or update the values incase of a future change/update.
Something like ...
class PodStatusPhase(str,Enum)
RUNNING = "Running"
PENDING = "Pending"
SUCCEEDED = "Succeeded"
FAILED = "Failed"
UNKNOWN = "Unknown"
NONE = "None" # adjust to what None is here
Then when returning the outputs from k8s,
if result:
try:
return PodStatusPhase(status_str)
except ValueError:
# Update this to handle what states would error
return PodStatusPhase.UNKNOWN
return None
And whenever in your code you are using the values you can easily use the values i.e
PodStatusPhase.FAILED
Contributor
Author
There was a problem hiding this comment.
Thanks for this Wanjiru ,this will move into a new task to be in another sprint.
… parse the env variables
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Related Issue (optional)
How to test this PR?
Screenshots / Logs (optional)
Checklist